Finite-Sample Analysis of Two-Time-Scale Natural Actor–Critic Algorithm

Actor–critic style two-time-scale algorithms are one of the most popular methods in reinforcement learning, and have seen great empirical success. However, their performance is not completely understood theoretically. In this article, we characterize global convergence an online natural actor–critic algorithm tabular setting using a single trajectory samples. Our analysis applies to very general settings, as only assume ergodicity underlying Markov decision process. order ensure enough exploration, employ $\epsilon$ -greedy sampling trajectory. For fixed small exploration parameter , show that has rate notation="LaTeX">$\tilde{\mathcal {O}}(1/T^{1/4})$ where notation="LaTeX">$T$ number samples, leads sample complexity {O}}(1/\delta ^{8})$ samples find policy within error notation="LaTeX">$\delta$ from global optimum. Moreover, by carefully decreasing iterations proceed, present improved ^{6})$ for